MapReduce is a prominent paradigm for big data analytics, and many systems support MapReduce jobs or tasks. MapReduce jobs have different phases that include, for example, map, copy and/or shuffle, merge and/or sort, and reduce. Resource consumption at different phases for different jobs has different characteristics. For example, certain phases can be characterized as central processing unit (CPU) heavy while other phases can be characterized as memory heavy or input/output (I/O) heavy.
However, existing MapReduce systems only allow static allocation of resources at initialization. Further, such approaches cannot adapt to changing resource demand, which can lead to inefficiency and sub-optimal job performance. Accordingly, a need exists for techniques for dynamic resource allocation in MapReduce.